Apparatus and method for rapid microscopic image focusing

ABSTRACT

A method of capturing a focused image of a continuously moving slide/objective arrangement is provided. A frame grabber device is triggered to capture an image of the slide through an objective at a first focus level as the slide continuously moves laterally relative to the objective. Alternatingly with triggering the frame grabber device, the objective is triggered to move to a second focus level after capture of the image of the slide. The objective moves in discrete steps, oscillating between minimum and maximum focus levels. The frame grabber device is triggered at a frequency as the slide continuously moves laterally relative to the objective so multiple images at different focus levels overlap, whereby a slide portion is common to each. The image having the maximum contrast value within overlapping images represents an optimum focus level for the slide portion, and thus the focused image. Associated apparatuses and methods are also provided.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional application of U.S. application Ser.No. 11/425,605, filed Jun. 21, 2006, now U.S. Pat. No. 7,417,213 whichclaims the benefit of U.S. Provisional Application No. 60/692,761, filedJun. 22, 2005, which are incorporated by reference herein in theirentirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to apparatuses and methods for the rapidfocusing and acquisition of images of objects and areas of interest on amicroscope slide.

2. Description of Related Art

Microscopic analysis is a widely used tool for research and routineevaluations, particularly in the field of cellular biology, cytology andpathology. Tissue samples and cell preparations are visually inspectedby pathologists under several different conditions and test procedures,through the use of a microscope. Based on such a visual inspection bythe pathologist, determinations concerning the tissue or cellularmaterial can be made. For example, in cancer detection and research,microscopic analysis aids in the detection and quantification of geneticalterations and/or anomalies that appear related to the cause andprogression of cancer, such as changes of expression of specific genesin form of DNA or messenger RNA (gene amplification, gene deletion, genemutation), or the encoded protein expression. Thesealterations/anomalies can either be assessed in microscopic slidesspecifically prepared to present individual cells, as is the standardprocedure in cytology, or whole histological sections or Tissue MicroArrays can be evaluated.

Although numerous other suitable laboratory techniques or analysesexist, microscopy is routinely used because it is an informativetechnique, allowing rapid investigations at the cellular andsub-cellular levels, while capable of being expeditiously implemented ata relatively low cost. However, in order to overcome, for example,subjectivity and/or repeatability limitations of conventionalmicroscopy, improved analysis devices combined the microscope withautomatic image analysis provisions. Such improved devices include, forexample, interactive systems, automatic scanning devices, and virtualslide scanners.

Interactive systems usually don't change the workflow of the pathologistanalyzing and interpreting slides underneath the microscope. Typically,such interactive systems allow the potential for extracting additionalquantitative information from the slide via image analysis and,therefore, possibly improve the reproducibility and the interpretationresults of the operator. Better tools for reporting and documentinganalysis results may also be realized. If properly configured,interactive systems are relatively fast and cost efficient, but theimpact of such interactive systems on routine workflow is relativelysmall.

Automatic rare event detection devices are typically set up in a waythat the whole analysis of the slides is done by the system in a totallyunsupervised manner, from the loading of the slides onto the scanningstage to the final reporting of the results. Such automatic systemsusually scan the slides, automatically identify objects or areas ofinterest for the analysis, quantitatively assess the targets, and reportand document the results. The routine workflow for the pathologist orcytotechnologist in general is changed drastically, from alabor-intensive screening task to the interpretation of analysisresults. However, such automatic systems are normally quite expensive,so that a relatively high annual volume of slides must be processed tocost-justify the acquisition of such a device.

Virtual slide scanning systems have been developed to automaticallyacquire large overview images at different optical resolutions. Suchoverview images can be far larger than the individual FOVs as they canbe seen in the microscope.

One common factor relating these three applications mentioned above,namely interactive systems, automatic scanning devices, and virtualslide scanners, is that each requires a specific focusing technique forfocusing an image, such as a digital image. For interactiveapplications, the operator usually manually focuses the image by visualinspection using the fine and coarse focus knobs of the microscope.However, in some instances, it is possible to implement a moresophisticated approach based on an auto-focus algorithm using the cameraand the motorized Z drive of the microscope. In the interactive mode,the constraints are relatively small since the image to be focused isstatic and the speed at which the image must be focused is not critical,unless the focusing is a background process. Different methods areavailable to implement generic auto-focus algorithms, such as, forexample, z-stacking or hill-climbing. These methods are based on theselection of the image presenting the highest contrast, where digitaloperators such as the variance, the entropy, and the LaPlacian, evaluatethe image contrast to determine the highest contrast for optimum focus.

Another method of operating the imaging system is the automatic rareevent detection mode. First, a slide is automatically moved to amotorized stage via a slide handler and a bar code on the slide is readby a bar code reader. Any objects of interest are automaticallyidentified based on a predefined criteria and a low-resolutioncontinuous motion scan of the region of interest (ROI) on the slide. TheROI can be determined based on a priori knowledge and is typically apart of a slide or a specific cell deposition area, defined through apreparation process (e.g. a liquid based preparation), or the ROI can bethe whole slide. Objects identified during the first scan are thenautomatically re-located on the slide and respective images thereofacquired at high resolution. The high resolution image(s) can then bedisplayed in an image gallery for local or remote pathologist review.

For such rare event detection mode, the speed at which the slide isscanned and objects of interest re-located is critical to an effectivesystem. When a low power objective is used for low-resolution scanning(i.e. 5×/0.15 NA) auto-focusing may not be necessary since thedepth-of-field at such low resolution is large enough to include thefocus plane of the specimen. However, the slide tilt must be evaluatedand compensated for during the scanning. When higher power objectives(i.e. 10×/0.3 NA or 20×/0.5 NA) are used, the depth-of-field is small,and any acquired digital images must include a pre-focusing of the fieldof view. A z-stacking or hill-climbing approach is possible only whenthe scanning of the slide involves a stop-and-go mode of operation. Thatis, the system must stop the scan, obtain a focus and acquire the image,and then restart the scan. This mode of operation leads to impracticallylarge scanning times. A progressive scan mode may be one alternative togain slide scanning speed, but such a mode requires a specific focusingstrategy capable of cooperating and functioning with the continuousscanning motion.

In the re-location mode for regions or objects of interest (for example,particular fields of view or individual objects, e.g. cells) a fastauto-focus is generally required. However, for auto-focusing onindividual objects or regions, known z-stacking or hill-climbingtechniques may be sufficient. The automatic re-location of detectedobjects of interest at high resolution requires efficient focusing inorder to present useful images in the image gallery for local or remotereview. Certain types of applications require the recapture of numerousobjects of interest and, as such, the amount of time spent for this taskis critical. Z-stacking or hill-climbing focusing methods must thereforebe optimized to reduce the time spent during recapture if such focusingmethods are to be sufficient.

The virtual slide scan mode relates to the acquisition and quantitativeevaluation of ROI's, which are larger than individual FOV's, and mayinclude the complete slide. The time constraints in this mode aregenerally the same, if not higher than, the rare event detectionscanning mode. Therefore, a progressive scanning approach with anappropriate focusing method may be best suited for such a situation.

In this regard, U.S. Pat. No. 5,912,699 to Hayenga et al. discloses amethod and apparatus for rapid capture of focused microscopic images,whereby specimen focus evaluation is conducted in a continuous scanmotion. The Hayenga device is equipped with a camera assembly having 3camera paths (primary camera, first and second focus camera) and a focusprocessor that evaluates the image focus in approximately real time atany position along the slide scanning path. The focus processorcalculates a score based on the differential ratio ((F⁻−F⁺)/(F⁻+F⁺)),where F⁻ and F⁺ are, respectively, the contrast evaluations of the imagegrabbed by the first and second focus cameras. However, such a cameraassembly is optimized for a particular magnification and cannot readilybe used for a different magnification setting.

U.S. Pat. No. 6,640,014 to Price et al. discloses a method ofsimultaneous multi-planar image acquisition. Particularly, an image ofthe specimen is captured as a three-dimensional (3D) volume, using anarray of 9 TDI line scan cameras connected to the microscope via fiberoptics. The focus is dynamically calculated, while a piezo-focus deviceupdates the focus position according to a tracking algorithm. However,such a method presents two potential drawbacks: (i) the equipment coststend to be very expensive; and (ii) the size of the 3D volume of theimage (number of planes, distance between the planes) may be limited byhardware constraints.

U.S. Patent Application Publication No. US 2004/0223632A1 to Olszacdiscloses a method of best-focus evaluation in continuous motion slidescanning. The image sensor (array of lenses or an array of cameras) istilted from the optical axis perpendicularly to the scanning direction(i.e. lateral scanning). At a given moment, the entire image sensor seesa scene (i.e. microscopic image) in which only the central line is infocus (assuming the device at mid-focus range) and the other linescorrespond to focuses above and below the best-focus line. The positionof the best-focus line will change as the specimen is scanned. Themethod thus described in this reference is used for pre-scanning theslide and basically serves as a focus map.

U.S. Patent Application Publication No. US 2004/0218263A1 to Brugaldiscloses a device for digital microscopy including a CMOS camera, apiezo-objective, a motorized stage, and a linear objective turret.However, no use of the system in a continuous motion image acquisitionprocedure is disclosed.

Thus, there exists a need for a method and apparatus for rapid focusingof an imaging system for obtaining a microscopic image, which can beused for static high-resolution object recapture as well as forcontinuous motion high resolution image focusing. Such an apparatus andmethod should desirably be relatively cost effective, have relativelylittle and/or simple equipment requirements, and be readily adaptable tovarious magnifications. In addition, the size of the image that can beobtained preferably should not be limited by hardware constraints. Sucha rapid focusing methodology should also be readily adaptable to asample exhibiting different focal planes, as well as to focusingconsiderations encountered in image analyses implementing chromogenseparation techniques.

BRIEF SUMMARY OF THE INVENTION

The above and other needs are met by the present invention which, in oneembodiment, provides a method of capturing a focused image, through anobjective, of a slide on a stage. At least one of the objective and theslide is configured to continuously move with respect to the other. Sucha method comprises triggering a frame grabber device to capture an imageof a portion of the slide through the objective as the slide moveslaterally with respect to the objective, wherein the objective isdisposed at a first focus level with respect to the slide. Alternatinglywith the step of triggering a frame grabber device, the objective istriggered to move to a second focus level with respect to the slideafter the image of the portion of the slide has been captured. Theobjective is movable in discrete steps in a range of focus levels,oscillating back and forth between a minimum focus level and a maximumfocus level, wherein the range of focus levels includes the first andsecond focus levels. The frame grabber device is triggered at afrequency as the slide moves laterally with respect to the objective sothat multiple images at different focus levels overlap and such that theportion of the slide is common to each of the multiple images, whereineach image has a contrast value. The image having a maximum contrastvalue within the plurality of overlapping images is then determined,with the maximum contrast image thereby representing an optimum focuslevel for the portion of the slide, and thus the focused image.

Another aspect of the present invention comprises a method of forming afocused image of a slide on a stage. Such a method includes identifyingobjects of interest in a specified-area image of a slide, wherein thespecified-area image have an image-wide maximum contrast correspondingto a image-wide focus level for the specified-area image, and thensorting the objects of interest according to a criteria. A localcontrast evaluation for each of the sorted objects of interest isperformed from local images of the object of interest captured atdiscrete focus levels about the image-wide maximum contrast focus levelso as to form an image stack index for each of the sorted objects ofinterest. Each image stack index includes one of the local images at afocus level corresponding to a local maximum contrast for the respectiveobject of interest. A pixel from the local maximum contrast local imageof one of the sorted objects of interest is substituted for thecorresponding pixel in the specified-area image having the image-widecontrast level so as to form a fused image, wherein the fused image isthereby configured to bring each of the sorted objects of interest inthe specified-area image into focus. A low-pass filter is then appliedto the fused image to reduce any local step effects around any of thesorted objects of interest brought into focus.

Yet another aspect of the present invention comprises a method ofscanning a slide so as to form a virtual image thereof. Such a methodincludes continuously moving one of a slide and an objective along apath in a first direction past the other of the slide and the objective,wherein the slide is supported by a stage. A focus map along the path isformed by focusing the objective with respect to each of a series ofportions of the slide disposed along the path by 1) triggering a framegrabber device to capture an image of one of the series of portions ofthe slide through the objective as the slide moves laterally withrespect to the objective, wherein the objective is disposed at a firstfocus level with respect to the slide; 2) alternatingly with the step oftriggering a frame grabber device, triggering the objective to move to asecond focus level with respect to the slide after the image of the oneof the series of portions of the slide has been captured, with theobjective being movable in discrete steps in a range of focus levels,and oscillating back and forth between a minimum focus level and amaximum focus level, wherein the range of focus levels including thefirst and second focus levels, and the frame grabber device is triggeredat a frequency as the slide moves laterally with respect to theobjective so that multiple images at different focus levels overlap andsuch that the one of the series of portions of the slide is common toeach of the multiple images, with each image having a contrast value;and 3) determining a maximum contrast value within the plurality ofoverlapping images, with the maximum contrast value thereby representingan optimum focus level for the one of the series of portions of theslide. The one of the slide and the objective is then continuously movedalong the path in a second direction past the other of the slide and theobjective, wherein the second direction is opposite to the firstdirection. For each of the series of portions of the slide along thepath, the objective is moved to the optimum focus level determined forthat one of the series of portions of the slide and included in thefocus map along the path. Once the objective is disposed at the optimumfocus level for that one of the series of portions of the slide, afocused image thereof is captured through the objective.

Still another aspect of the present invention comprises a method ofcapturing a focused image, through an objective, of a slide on a stage,wherein at least one of the objective and the slide is configured tocontinuously move with respect to the other. Such a method includescontinuously moving one of a slide and an objective in a direction pastthe other of the slide and the objective, wherein the slide is supportedby a stage. At least one image of a portion of the slide is capturedthrough the objective with a focus imaging device, as the slide moveslaterally with respect to the objective, at a first focus level above afocal plane and at a second level below a focal plane with respect tothe slide. A maximum contrast value is determined from the at least oneimage, with the maximum contrast image thereby representing an optimumfocus level for the portion of the slide. A focused image of the portionof the slide is then captured through the objective, with a slideimaging device disposed at least one field of view behind the focusimaging device in the movement direction, as the slide moves laterallywith respect to the objective, at the optimum focus level determinedfrom the images captured by the focus imaging device.

Yet another aspect of the present invention comprises a method ofcapturing a focused image, through an objective, of a slide on a stage.Such a method capturing, with an imaging device, an image of a portionof a sample on the slide, through the objective, at each of a pluralityof focus levels about a focal plane with respect to the sample, with thesample being treated with a plurality of dyes. A chromogen separationprocedure is then performed on the sample portion image at each of theplurality of focus levels, and a dye space image of the sample portionimage is formed at each of the plurality of focus levels for each of theplurality of dyes. For each of the plurality of dyes, the dye spaceimage having a maximum contrast is then selected, with the maximumcontrast dye space image thereby representing an optimum focus level forthe respective dye in the sample portion image. The optimum focus leveldye space images for each of the plurality of dyes are then combined toform an optimum focus sample portion image.

Further, another aspect of the present invention comprises a method ofcapturing a focused image, through an objective, of a slide on a stage.Such a method includes capturing, with an imaging device, an image of aportion of a sample on the slide, through the objective, at each of aplurality of focus levels about a focal plane with respect to thesample, with the image having a plurality of pixels. A maximum contrastpixel is then selected from the corresponding pixels in the sampleportion images at the plurality of focus levels, with the maximumcontrast pixel thereby representing an optimum focus level for therespective pixel of the sample portion image. The maximum contrastpixels are then combined to form an optimum focus sample portion image.

Suitable apparatuses for implementing and accomplishing the disclosedmethods are also provided, wherein many different apparatusconfigurations and arrangements may be used, as will be appreciated byone skilled in the art. Accordingly, embodiments of the presentinvention provide distinct advantages as described and further discussedherein.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

Having thus described the invention in general terms, reference will nowbe made to the accompanying drawings, which are not necessarily drawn toscale, and wherein:

FIG. 1 schematically illustrates an apparatus for capturing a focusedimage, through an objective, of a slide on a continuously moving stage,according to one embodiment of the present invention;

FIG. 2 schematically illustrates one configuration of an apparatus forcapturing a focused image, through an objective, of a slide on acontinuously moving stage, according to one embodiment of the presentinvention;

FIG. 3 schematically illustrates a method of extracting a highestcontrast image from a stack of images obtained at a series of differentfocus levels, according to one embodiment of the present invention;

FIG. 4 is a flowchart of a method of extracting a highest contrast imagefrom a stack of images obtained at a series of different focus levels inan image recapture process, according to one embodiment of the presentinvention;

FIG. 5 schematically illustrates a method of slide scanning by band orstripe, according to one embodiment of the present invention;

FIG. 6 schematically illustrates a method of triggering objective focusand image capture processes for capturing a focused image, through anobjective, of a slide on a continuously moving stage, according to oneembodiment of the present invention;

FIG. 7 schematically illustrates a method of scanning portions of aslide through capture of a stack of images at different focus levels,through an objective, with the slide being on a continuously movingstage, according to one embodiment of the present invention;

FIG. 8A schematically illustrates a method of evaluating the highestcontrast level in each stack of images captured through a scanningprocess disclosed in FIG. 7, and reconstructing a composite focusedimage of the slide from the highest contrast images, according to oneembodiment of the present invention;

FIG. 8B illustrates one example of the difference in focus on amonolayer PAP cytology specimen within the same field of view,particularly experienced when observing the field of view with a highnumerical aperture (i.e., N.A.>0.5);

FIG. 8C illustrates a method of optimizing the contrast stack evaluationby detecting out-of-focus objects of interest, evaluating localcontrast, and then forming a fused image;

FIG. 9 schematically illustrates a method of forming a focus map of aportion of a slide using a method of triggering objective focus andimage capture processes, through an objective, of a slide on acontinuously moving stage, according to the embodiment of the presentinvention disclosed in FIG. 6;

FIG. 10 schematically illustrates a method of scanning a slide through adual pass scanning procedure comprising determining a focus map of theslide in one direction along a slide path as disclosed in FIG. 9 andthen capturing images of the slide according to the focus map in thereverse direction along the slide path;

FIG. 11 schematically illustrates an apparatus for capturing a pair ofimages, both above and below a focal plane, in order to implement amethod of determining an optimum focus, through an objective, of a slideon a continuously moving stage, according to another embodiment of thepresent invention;

FIG. 12 schematically illustrates a method of capturing a focused image,through an objective, of a slide on a continuously moving stage bydetermining an optimum focus with a focus imaging device as disclosed inFIG. 11 and then capturing a focused image at the optimum focus with aslide imaging device disposed behind the focus imaging device in thescan direction, according to one embodiment of the present invention;

FIG. 13 includes images of the above and below focal plane images, andthe focused image at the optimum focus, using an apparatus as disclosedin FIG. 11 and a method as disclosed in FIG. 12, according to oneembodiment of the present invention;

FIG. 14 schematically illustrates an apparatus implementing a primarycamera and a focus camera, for accomplishing a method as disclosed inFIG. 12, according to one embodiment of the present invention;

FIG. 15 schematically illustrates a relationship between the above andbelow focal plane images and image contrast, as a function of the focuslevel of the objective with respect to the slide, according to oneembodiment of the present invention;

FIG. 16 schematically illustrates multi-plane focus requirements due todifferent interactions between different dyes and cells of interest;

FIG. 17 schematically illustrates a method of chromogen separationwhereby two dye images are created from an RGB color image;

FIGS. 18A and 18B schematically illustrate a method for evaluating anoptimum focus in each of the dye spaces, determined according to achromogen separation method as shown in FIG. 17 and at different focuslevels in each dye space (FIG. 18A), according to one embodiment of thepresent invention, whereby optimum focus level images in each dye spacecan be combined to produce a best focus image (FIG. 18B);

FIG. 19 schematically illustrates a method of combining the focusedimages by digitally extending the depth-of-field of a stack of images,wherein two consecutive frames of the same field at different closefocus values are combined in accordance with a decision rule whether todetect and keep in the final image the maximum of contrast in each pixelof the image; and

FIG. 20 schematically illustrates a method of combining the focusedimages, as shown in FIG. 19, using a digital wavelet transform.

DETAILED DESCRIPTION OF THE INVENTION

The present inventions now will be described more fully hereinafter withreference to the accompanying drawings, in which some, but not allembodiments of the inventions are shown. Indeed, these inventions may beembodied in many different forms and should not be construed as limitedto the embodiments set forth herein; rather, these embodiments areprovided so that this disclosure will satisfy applicable legalrequirements. Like numbers refer to like elements throughout.

I Oscillating Focusing

Embodiments of the present invention are directed to apparatuses andmethods for rapid microscopic image focusing that can be used for bothstatic high-resolution object recapture and for continuous motionhigh-resolution image focusing.

One embodiment of the present invention implements an integrated system50 as shown, for example, in FIG. 1, that includes a microscope 100 withbuilt-in automation functionalities (e.g., a Zeiss Axiolmager), amotorized stage 150 for the microscope 100, a progressive area scancamera 200 (e.g., a CCD or a CMOS type camera), and a fast focus device250 (e.g., a PIFOC piezo-objective from Physik Instrument, Germany) incommunication with a controller 275. The motorized microscope stage 150is in communication with a computer device 300 via a controller 350,while the camera 200 is also in communication with the computer device300 via a controller, such as a frame-grabber device, wherein, in oneembodiment, such a controller may be integrated or combined with thecontroller 275 for the fast focus device 250, as shown in FIG. 1. Oneskilled in the art will appreciate, however, that the objective of themicroscope 100 may be configured to move with respect to a stationarystage 150 and/or the slide to be examined may be configured to bemovable with respect to a stationary objective and, as such, theconfigurations of an integrated system 50 disclosed herein are merelyexamples of possible configurations thereof.

The progressive area scan camera 200 is, for example, a color cameracapable of acquiring a full image at a time so that image jitter formoving objects is substantially eliminated. Such a camera 200 generallyincludes an integrated shutter function, which allows electronicadjustment of exposure times within a wide range, which allows themovement of a passing object to be optically frozen without expensivestrobe illumination. In order to optimize the progressive scan, thecamera 200 may be configured to run an asynchronous reset mode or anyother suitable mode capable of optimizing the progressive scanprocedure.

In one embodiment, the fast focus device 250, such as a piezo-objective,can be, for example, a microscope objective nano-focusing device withtravel ranges from between about 100 μm and about 500 μm, withsub-nanometer resolution. Such a device is screwed on the objectiveturret in a regular objective slot. The objective is then screwed on theobjective holder of the nano-focusing device. Such nano-focusing devicesare relatively faster and more accurate than regular focus devices (50μm step—1% accuracy—in less than 10 ms). There are several ways tocontrol (move up/down) such a piezo-objective. One way is to send aRS-232 or USB command to the controller 275 to move the piezo-objectiveup or down. Sending and interpreting the command can take, for example,several milliseconds. However, in some instances, such a method ofcontrol may be too slow and cannot be effectively used.

Accordingly, one embodiment of the present invention implements aparticular electronic controller to control and activate thepiezo-objective with the necessary speed to provide efficient focusingduring the scanning procedure, while allowing two modes of operation.Though one configuration is described herein, one skilled in the artwill appreciate that the functions of such a controller may beaccomplished in many different manners, and the configuration disclosedherein is but one such example of an appropriate configuration.

In a static re-location mode, a Digital to Analog Converter (DAC) typecontroller 275 controlled by the computer device 300 forces thepiezo-objective to run a continuous triangular or other continuoussinusoidal wave. The DAC controller 275 determines the shape, thefrequency and the dynamics of the wave. At pre-defined voltage valuesalong that wave, the DAC controller 275 pulses out TTL triggers to startthe image acquisition, whereby images are acquired by the camera 200through a direct connection to the frame-grabber device. Moreparticularly, one embodiment of such an electronic controller 275 is amodified sound card of the computer device 300 that acts as a digital toanalog converter (DAC) and an amplification layer to match the requiredvoltage (see, e.g., FIG. 2). The amplification layer of the electroniccontroller 275 has 2 independent channels that are connectedrespectively to the piezo-objective (DAC) and the frame-grabber/camera200 (TTL trigger out). The channel of the electronic controller 275connected to the piezo-objective generates oscillations (wave forms)that can be adjusted in frequency and amplitude by sending appropriatesoftware commands to the sound card (i.e., MS Windows Direct Sound API).The channel of the electronic controller 275 connected to theframe-grabber/camera 200 generates compatible TTL triggers (N=5, 9 or17) with a lock up (synchronization) mechanism. For example, when N=9, 9pulses are sent to the frame-grabber/camera 200 as the piezo-objectiveis moved according to half of a wave cycle (min to max). The amplitudeis determined according to the depth-of-field of the particularmagnifying objective in use and the desired focusing accuracy.

In a dynamic focusing mode (i.e., focusing the piezo-objective duringcontinuous stage motion), the controller 275, is set by the RS232 or USBconnection, in slave mode. N (N=5, 9, 17) predefined values are disposedon a triangular wave. As such, each increment of the piezo-objective tothe next value on the wave is operated by a TTL trigger IN signal sentby the frame grabber after the previous image acquisition by the camera200. The frame-grabber triggers the image acquisition by the camera 200when receiving a TTL trigger IN signal from the stage 150 running incontinuous motion. The wave amplitude and the number of predefinedvalues are adjusted according to the depth-of-field of the particularmagnifying objective in use and the desired focusing accuracy. The stage150 (master), the frame-grabber/camera 200 (slave) and thepiezo-objective (slave) are synchronized independently of the speed ofmotion of the stage 150, as long as the piezo-objective and cameraframe-rate are compatible with the triggering frequency.

I.1 High Resolution Object Recapture

In the static mode of operation (high resolution object recapture), themotorized stage 150 moves to the X, Y coordinates of a selected object,while the built-in fast focus device 250 moves the piezo-objective to astored position corresponding to the particular magnifying objective inuse. Once the X, Y, and Z axes are stabilized, the system 50 can collecta dynamic stack of images that are acquired by the camera 200 at N(i.e., N=10) times the frequency of the piezo-objective oscillation, andthe focusing oscillation is not stopped as the stage moves from oneposition to another.

For each image of the stack, a digital operator such as the variance isused to evaluate the contrast, as shown in FIG. 3. The image exhibitingthe highest value of contrast is kept as the best focus image at the X,Y position. When running the piezo-objective at a frequency of about 5Hz, the camera 200 grabs images at a frequency of about 50 Hz. The focusis thereby evaluated out of 10 images by the system 50 as fast as 50frames/sec (1 full cycle=200 msec). In this mode, the exact value ofcontrast along the focus axis (Z axis) is not particularly necessarysince obtaining only the best-focused image is of interest. Thedepth-of-field of a 20× objective (NA 0.5) equals about 2.5 μm and, atthis magnification, a z-stack of 15 μm (10 images, 1.5 μm step size) istypically used. If the optimum focal plane is not “bracketed” by thez-stack (i.e., if the optimal focal plane does not fall within themagnitude of the range of focus levels, either by the relative contrastevaluation or through a threshold determination), the highest contrastimage of the first and the last image index (0 or N) of the particularz-stack is determined. In such a situation, the initially storedZ-position for the microscope objective is too far off the focal planeat the selected X, Y position (i.e., slide tilt not correctlycompensated). As such, the system 50 increments the built-in fast focusdevice 250 position by the size or magnitude of the z-stack (i.e.,shifts the fast focus device 250 position by the magnitude of the focalrange) in the direction of the previous highest contrast image (index 0or N), wherein the piezo-objective is not stopped during the Z-positionrepositioning. As soon as the Z-axis (the built-in fast focus device250) is stabilized, the highest-contrast image is extracted from thenewly-obtained z-stack, as shown in FIG. 4.

I.2 Direct High Resolution Scanning

In the continuous motion mode, the motorized stage 150 scans the slideby the camera 200/piezo-objective in a pattern of sequential adjacentstripes. The number of bands depends on the size of the camera sensorand the overlap between the stripes required by the application, asshown in FIG. 5. Before the acquisition of a stripe, the oscillation ofthe piezo-objective is initiated in accordance with the triangular wave,and the camera 200 acquires images with the scheme described in FIG. 6.During the stripe acquisition, the motorized stage 150 moves along oneaxis (i.e., the X-axis) at a given speed, and the stage controller 350delivers pulses (i.e., TTL triggers) to the frame-grabber at predefinedintervals. As a consequence, the system 50 accumulates image stacks witha fixed offset between the images in a particular stack. Because of theoffset between the images, the contrast cannot be compared from oneimage (or plane) to another on the entire image. That is, only theoverlapped part of the images can be used for focusing, as shown in FIG.7. The offset is dependent on the size of the stack (i.e., the number ofplanes) and the size of the camera sensor.

When the offset is adjusted so that

${{Offset} = \frac{Fieldsize}{2*\left( {{Stacksize} - 1} \right)}},$a complete stack containing a half field-size image is available forfocus evaluation every half field. A series of adjacent stacks is thenprocessed so that the best-focused image is extracted from each stackaccording to the same contrast detection algorithm described earlier(i.e., variance, first-derivative, local contrast). The adjacentbest-focused images can subsequently be tiled together to obtain oneunique digital image of the given stripe (i.e., a virtual slide). Theseries of focused images can also be used in rare event detection forindividual field of view data extraction without the tiling operation.

The width of each stripe is defined by the Y-dimension (D_(y)) of thesensor portion of the camera 200, the magnification factor (M) of theselected microscope optics creating the analog image on the camerasensor, and a chosen overlap area (O_(y)) between two adjacent bandsnecessary for the correct alignment of the bands to form a completeimage. The X-direction is defined, in this example, as the direction inwhich the stage 150 moves during the scanning process of acquiringimages to create a complete stripe, while the Y-direction is orthogonalto the X-direction within the object plane. As such, given the scan areaS with the dimensions x and y:S=x·yand the number of bands (N_(stripe)) needed to cover S is defined as:

$N_{stripe} = \frac{M \cdot y}{D_{y} \cdot \left( {1 - \frac{O_{y}}{N_{y}}} \right)}$

Using an example of a scan implementing an objective with magnificationM=20 to cover a scan area with dimensions x=13.5 mm and y=13.5 mm with atypical overlap between adjacent bands of O_(y)=40 pixels and a ⅔″ 3CCDprogressive scan camera 200 with chip (sensor) dimension in Y-directionD_(y)=7.2 mm and pixel resolution of N_(x)×N_(y)=1024×1024 pixel, thenumber of bands that must be computed is about 40.

Generally, any array-type camera 200 is suitable for implementation inthe method, though the number of bands and the overall time ofperformance of the method is directly affected by the size of the camerasensor. The stripes or bands are scanned following a unidirectionalpattern. During the acquisition of a stripe, the stage 150 moves atconstant speed V_(stage). The speed of the stage 150 is directlydependent on the X-dimension (D_(x)) of the camera sensor, themagnification (M), the depth of the stack (N_(image)), and the framecapture rate of the camera 200 (Camera_(rate)).

${Offset} = \frac{D_{x}}{M \cdot 2 \cdot N_{image}}$

Using the example of a scan with magnification M=20, N_(image)=7 imagesper stack, and a ⅔″ 3CCD progressive scan camera 200 with chip dimensionin the X-direction D_(x)=9.6 m, the offset equals 34 μm, and one imageis acquired at every offset in the X-dimension.V _(stage)=Offset·Camera_(rate)

At a camera frame-rate of 100 images/sec, the speed of the stage equals3.4 mm in the same time period (1 sec).

The overall time of scanning (T_(scan)) is calculated as followed:

$T_{scan} = {N_{stripe} \cdot \left( \frac{x}{V_{stage}} \right)}$

With the settings described in the above example, T_(scan)=154 seconds(<3 minutes). The processing time between 2 successive images is thus 10msec at 100 images/sec. This timing can be important for efficient realtime image processing (i.e., contrast evaluation, shading correction,etc.). Therefore, the stacks are processed in a separate thread with aslight delay (half a field) and immediately made available for the otherthread for image storage, image tiling or data extraction as required bythe application.

This described method can be used at many magnifications (i.e., 10×,20×, and 40×). Generally, the piezo-objective is attached to a singleobjective at a time. As such, if the application requires the method tobe run on 2 different magnifications (i.e., 10× and 20×), the opticalpath can be equipped with a zoom lens (for example, Zeiss Optovar) sothat an equivalent 20× magnification is obtained by using simultaneouslythe 10× objective and 2× zoom.

The contrast evaluation is processed for each image in the stack on theoverlapped portion of the images (i.e., half field). For extendedaccuracy, the stack can be subdivided so that the contrast is evaluatedon a sub-image, as shown in FIG. 8A, without changing the acquisitionprocess. Such a process makes it possible to display or process focusedobjects of interest belonging to the same field of view (half the fieldof view seen by the camera 200) even if such objects were not sharingthe focal plane (See FIG. 7). When sub-images are used, the contrastevaluation procedure must implement, for example, a low-pass filter, inorder to reduce “step effects” between sub-images. That is, in monolayercytology slide scanning, a pseudo-3D effect in the fields of view may berealized when observing the slide at high numerical aperture (i.e.,N.A.>0.5) due to the small depth of field. This pseudo-3D effect leadsto a situation where it may not be possible to have the entirety of theobject of interest focused in the field of view (see, e.g., FIG. 8B).The contrast evaluation is thus performed on the entire field of view soas to provide an image at an “average” focal plane.

In a rare event detection application (i.e., Pap scanning), only asubset of the cells within the sample are of interest. As such, in suchapplications, in order to enhance the focusing accuracy, a procedure asshown, for example, in FIG. 8C, may be applied to bring some of thecells of interest into focus with the remainder of the field of view.More particularly, once a stack has been grabbed, the global contrast isevaluated and the corresponding image index in the stack is identified.A threshold segmentation procedure is then applied to separate thebackground of the image from the object of interest. An image labelingprocedure is then applied to sort objects of interest according tovarious criteria such as, for example, size and shape factor. Throughcorresponding rectangles (i.e., left, top, right, bottom objectcoordinates), local contrasts are evaluated for each object of interestselected and retained by the image labeling procedure. The localcontrast evaluation is associated with an image stack index, and may bedifferent from the global contrast evaluation. The final image for thecurrent field of view is then built by fusing pixels from the imageselected by the global contrast evaluation and the pixels determinedlocally through the local contrast evaluation, with a low-pass filterbeing applied to reduce any local step effects around any object ofinterest brought back in focus through such a procedure. In someinstances, such a procedure can be combined, for example, a chromogenseparation procedure (see, e.g., U.S. Patent Application PublicationNos. US 2003/0138140 and/or US 2003/0091221, each to Marcelpoil et al.)to allow certain types of cells (i.e., DAB marked cells as opposed tounmarked cells) to be selected with increased accuracy.

I.3 Dual-Pass High Resolution Scanning

Another way to use the apparatus 50 is to adjust the offset so that

${Offset} = \frac{Fieldsize}{Stacksize}$

In such a case, a complete stack is available for focus evaluation everyfield of view. However, the images are only equal to the size of theoffset, and there are gaps between the stacks, as shown in FIG. 9.Accordingly, this configuration may not be suitable for direct highresolution scanning as described above.

However, this configuration can first be used to perform a first scan tobuild a dynamic focus map along the main axis of the stripe. This focusmap is then used to perform a regular progressive scan with focusadaptation in a reverse direction along the same stripe, as shown inFIG. 10. This approach efficiently replaces and provides a significantimprovement over a method of placing random seeds all over the specimento be scanned and building a static 3D focus map with classicalinterpolation methods.

In this configuration, the Z-value corresponding to the highest-contrastin the stack is extracted, but this Z-value does not necessarilycorrespond to one of the images of the stack. For example, aninterpolation method (i.e. spline) can be used to fine-estimate thecontrast maximum. Since there is no image extracted out-of-the stack,the camera 200 can run with a lower spatial resolution and with alimited number of colors. For higher accuracy requirements, the dynamicfocus map can be combined with the first direct high resolution scanningmethod previously described.

II Predictive Dual Mirror Auto-Focusing

Another embodiment of the present invention implements a method using anintegrated system 50 comprising a microscope 100 with built-inautomation functionalities (for example, Zeiss Axiolmager), a motorizedstage 150, a progressive area scan camera 200 (CCD or CMOS), a fastfocus device 250 (i.e., PIFOC piezo-objective from Physik Instrument,Germany), and an optical device 450 (as shown in FIG. 11) having a tubemember (not shown) and an optical bench (not shown). The tube memberincludes an aperture 525 that serves as a field stop, and one or morelenses 550 for focusing and/or recollimating the incoming light. Theoptical bench supports a focus camera 600, a pair of beam-splitters 650a, 650 b, and a pair of mirrors 700 a, 700 b for splitting andrecombining the light, and lenses 750 a, 750 b for focusing the lightonto the CCD of the focus camera 600.

Generally, a single camera 200 and the optical device 450, such as theimage splitter mechanism described above, are used to obtain twoseparate images. Those two images, while covering the same field of vieware focused at different elevations (Z− and Z+). By comparing imagecontrast features, the elevation of best focus for the primary cameracan be determined. To make this focus evaluation compatible with acontinuous motion scanning approach, the focus camera 600 “looks ahead”in the scan direction at a field of view that has not yet been acquiredby the primary camera 200, as shown in FIG. 12. To do so, the aperture525 (see field stop of FIG. 11) is off-center by one magnified field ofview, so that the focus image covers a field of view adjacent to the onebeing acquired by the primary camera 200.

II.2 Hardware Description

In one embodiment, the required object size is at least 360 μm×480 μm,which corresponds, for example, to a ⅓″ CCD camera 200 shooting a sceneat 10× magnification. The focus system magnification is roughly 78% ofthe primary magnification. This is required in order to fit two imagesonto a single image sensor and still leave additional room on the sensorto accommodate any misregistration between the primary and focus images.Thus, in actuality, the focus magnification is about 7.8× when theprimary magnification is 10×.

An aperture 525 is required and acts as a field stop, preventing the twoimages from overlapping when focused onto the CCD image sensor. In orderto fit two images compatible with the ⅓″ format (3.6 mm×4.8 mm) of theprimary camera 200 onto the ½″ sensor (4.8 mm×6.4 mm) of the focuscamera 600, the magnification in the focus system must be reduced to nomore than about 88% of the primary system magnification. By doing so,and by using a field stop, each focus image is reduced to a width ofabout 3.2 mm, corresponding to an object width of 0.36 mm at 8.9×magnification. The image height may be as small as about 4.3 mm,corresponding to an object height of about 0.48 mm at 8.9×magnification. The length of the image may be greater to accommodatemisregistration between the primary and focus images in the Y-direction.In addition, the magnification may be less than 8.9× to allow formisregistration between the primary and focus images in the X-direction.The focus magnification is thus roughly 78% of the primarymagnification.

The processing zones of the two half images on the auto-focus camera 200must be adjusted, as shown in FIG. 13, to make sure that the two zonescorrespond to or “see” the same portion of the scene. In one embodiment,there is no magnification difference between the two half images, butonly a translation (X, Y) due to the misalignment of the mirrors alongthe two optical paths. This translation is estimated by performing anautocorrelation operation, which can be performed only if the two imagesshow approximately the same level of contrast. The primary camera isfirst focused and, once the translation (X. Y) is estimated, the twozones are optimized in size to cover the maximum surface of the scene.

The primary camera 200 is a color camera that grabs a true color imagefor further processing while the focus camera 600 is black & whitecamera for focus evaluation. Both cameras 200, 600 are configured tograb a field at the same time by, for example, connecting both cameras200, 600 to the same frame-grabber and then implementing a rapid channelswitching to alternatively grab each image. In the alternative, a dualframe-grabber architecture can be used, as shown in FIG. 14, where oneembodiment uses, for example, a Matrox Meteor II/MC for the primarycamera 200 and a Matrox Meteor II for the auto-focus camera 600. Thecameras 200, 600 are “gen-locked” to allow substantially exactsynchronous grabs. Gen-locking indicates that both cameras 200, 600share the same video standard (PAL or NTSC).

II.2 Algorithm Description

The instant focus algorithm is based on the assumption that thedifferential contrast is a linear function of Z between Z (F+) and Z(F−):

${F(z)} = \frac{\left( {{Cont}_{{ZF} +} - {Cont}_{{ZF} -}} \right)}{\left( {{Cont}_{{ZF} +} + {Cont}_{{ZF} -}} \right)}$

This function is linear between F+ and F−, as shown in FIG. 15, and thisrange is limited by the optical adjustment. One suitable range settingfor the 10× magnification is, for example, 40 μm. A larger range maylead to, for instance, some undesired non-linearity over the curvebetween F+ and F−. Because the two optical paths (F+ and F−) are notexactly identical (due to, for example, magnification, mirrororientation, misalignment, etc.), the two curves (F+ and F−) tend not tohave the same maximum, though this can be corrected by applying ashading correction.

Since the portion of the contrast function curve between F+ and F− isclose to be linear, the ultimate or optimal focal plane can be retrievedby applying the linear function (constant ˜0):Z _(Focus)=(Slope*Contrast)+Constant

In practice, the slope may be obtained through a pre calibration processon a slide test. Because of the inflexions on the maximum and minimum ofthe contrast function, the slope is calculated slightly inside therange, which slightly reduces the range of operation. However, theslightly reduced range of operation is compensated for by a moreaccurate focus estimation.

There are multiple methods for estimating the contrast of a scene using,for example, the variance, the histogram range, the entropy, theLaPlacian, etc. For example:

Variance:

${Var} = {\frac{1}{N}{\sum\left( {x - \overset{\_}{x}} \right)^{2}}}$First Derivative:

FD = ∑(x_(n) − x_(n + 1))

Because the focus position is calculated a field in advance during theband or stripe scanning procedure, the piezo-objective can be updatedwith the new focus position (a field in advance) while the stage 150 ismoving from one field to the next, while maintaining optimum focus alongthe band. A piezo-objective has been found to be very effective inallowing the focus to be performed sufficiently fast.

III. Chromogen Separated Autofocusing Method

In protein expression analyses, immunohistochemistry (“IHC”) andimmunocytochemistry (“ICC”) techniques, for example, may be used. IHC isthe application of immunochemistry to tissue sections, whereas ICC isthe application of immunochemistry to cultured cells or tissue imprintsafter they have undergone specific cytological preparations such as, forexample, liquid-based preparations. Immunochemistry is a family oftechniques based on the use of a specific antibody, wherein antibodiesare used to specifically target molecules inside or on the surface ofcells. The antibody typically contains a marker that will undergo abiochemical reaction, and thereby experience a change of color, uponencountering the targeted molecules. As such, chromogens of differentcolors can be used to distinguish among the different markers.

During relocation of an object of interest on a slide, variation offocal planes between dyes (Hematoxylin and DAB, for instance) can beobserved in the objects (i.e., cells) to be relocated as shown, forexample, in FIG. 16. In such an instance, the DAB is fixed to the cellthrough an enzymatic reaction when binding to the antibody of interest(i.e., protein). The result of this reaction is an enzymatic precipitatethat occurs on the top of the cell, while the counterstaining (i.e.,Hematoxylin) binds acidic components within (inside) the cell. Thedifference in focus between the DAB and Hematoxylin sites is on theorder of about 1 or 2 microns, and is generally visible only when thedepth-of-field of the objective is close to that measurement. This maybe the case when using a 20×/0.5 NA objective, for instance.

When a color CCD digital camera is used to image the sample, three graylevel images of the sample are simultaneously captured and obtained(each gray level image corresponds to the respective Red, Green and Bluechannel (RGB)). Chromogen separation techniques such as disclosed, forexample, in U.S. Patent Application Publication No. US 2003/0138140and/or US 2003/0091221, each to Marcelpoil et al. (see, e.g., FIG. 17),can then be applied to the image(s). As such, the optical density ofeach molecular species can be evaluated (as revealed by the chromogen ordye associated with each molecular species) in any location of theimage, generally on a per pixel level. On the biological sample, themarkers and counterstain generally indicate the dyes of interest to bedetected and quantified.

The concentration of the molecular specie can thus be determined from acolor image of the sample. In a video-microscopy system equipped with a3CCD camera, the image should generally be balanced and normalizedaccording to an empty field white reference and a black field image, andalso corrected for shading. In addition, the image is spatiallycorrected for chromatic aberrations, channel by channel. Once the imageis obtained, an optical density of the sample is computed in each of thered, green, and blue channels of the RGB image at a particular pixel inthe image from the measured light transmitted through the sample. Acorresponding optical density vector is thereafter formed for thatpixel. The optical density vector is then multiplied by the inverse of arelative absorption coefficient matrix of the dyes present in the sampleso as to form a resultant vector for the pixel, representing the opticaldensity contribution from each dye. The relative absorption coefficientmatrix comprises a relative absorption coefficient for each dye(marker(s) and counterstain(s)) used in the sample preparation protocol,in each of the red, green, and blue channels. The resultant vector thuscomprises the concentration of the molecular species, as indicated bythe respective marker(s), and by the counterstain(s), for that pixel.

Such imaging techniques, also known as multi-spectral imagingtechniques, when adapted to color imaging (RGB camera), allow asubstantially real time (video rate) processing of the sample(typically, for example, about 40 millisecond per frame), which providesan advantage. For speed issues and real time processing, or displayingpurposes in case of the use of an RGB camera, the acquisition throughthe different channels is performed in parallel and look up tables (LUT)can be generated which map the RGB color input values to pre-computedconcentrations and/or transmittances of each of the involved dyes.

Thus, another aspect of the present invention implements the focusingtechnique(s) disclosed herein to determine the optimum focal position orfocus level of both the marker and the counterstain (i.e., DAB andHematoxylin) resulting from chromogen separation techniques and tocombine together the two corresponding optimum focal planes of themarker and counterstain to obtain a unique optimum focus image of thesample, as shown in FIG. 17. In order to detect or determine the focalposition that corresponds to the highest contrast image out of a seriesof images, a hill-climbing or a z-stacking approach, for example, can beused. Once the optimum focus images of both the marker and counterstainare determined, the two focused images can be combined by summing, on aper pixel basis, the optical densities (OD) according the Lambert-Beerlaw, as shown in FIGS. 18A and 18B.

In the alternative, the combination of the focused images can beperformed using a method that digitally extends the depth-of-field of astack of images (“extended focus”), as shown, for example, in FIG. 19.More particularly, two consecutive frames of the same field at differentclose focus values can be combined or fused, wherein the fusion isperformed in accordance with a decision rule whether to detect and keepin the final image the maximum of contrast in each pixel of the image.One such implementation is based on a digital wavelet transform asshown, for example, in FIG. 20. In general, the offset (focusdifference) between the dyes (marker and counterstain) is almostconstant from one field of view to another. As such, once the focus isdetermined, the relocation process for objects of interest on a slidecan be accelerated, in some instances, by performing the optimum focusevaluation on one dye and then applying the focus offset to determinethe best focus image in the other dye of interest.

Many modifications and other embodiments of the inventions set forthherein will come to mind to one skilled in the art to which theseinventions pertain having the benefit of the teachings presented in theforegoing descriptions and the associated drawings. Therefore, it is tobe understood that the inventions are not to be limited to the specificembodiments disclosed and that modifications and other embodiments areintended to be included within the scope of the appended claims.Although specific terms are employed herein, they are used in a genericand descriptive sense only and not for purposes of limitation.

1. A method of forming a focused image of a slide on a stage,comprising: identifying objects of interest in a specified-area image ofa slide, the specified-area image having an image-wide maximum contrastcorresponding to an image-wide focus level for the specified-area image,and sorting the objects of interest according to a criteria; performinga local contrast evaluation for each of the sorted objects of interestfrom local images of the object of interest captured at discrete focuslevels about the image-wide maximum contrast focus level so as to forman image stack index for each of the sorted objects of interest, eachimage stack index including one of the local images at a focus levelcorresponding to a local maximum contrast for the respective object ofinterest; substituting a pixel from the local maximum contrast localimage of one of the sorted objects of interest for the correspondingpixel in the specified-area image having the image-wide contrast levelso as to form a fused image, the fused image thereby being configured tobring each of the sorted objects of interest in the specified-area imageinto focus; and applying a low-pass filter to the fused image to reduceany local step effects around any of the sorted objects of interestbrought into focus.
 2. A method according to claim 1 further comprising,prior to identifying objects of interest: capturing a plurality ofdiscrete focus level images of a portion of the slide within a range offocus levels; and determining the image-wide maximum contrast for theplurality of discrete focus level images corresponding to the image-widefocus level for the portion of the slide, so as to determine thespecified-area image.
 3. A method according to claim 2 furthercomprising applying a threshold segmentation criteria to thespecified-area image so as to separate a background therefrom.